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© Method and apparatus for displaying sign language images corresponding to text or speech. 

© In order to provide an image display method 
and apparatus capable of displaying images 
corresponding to document data or speech 
data, image data (such as sign language im- 
ages) are stored in an image dictionary in mo- 
tion picture form. Document data is read out 
from a character information storage device (or 
speech data is received), and a sign language 
image corresponding to a character string of 
the document data (or the speech) is selected 
from the image dictionary and displayed on a 
display. 
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The present invention relates to an image display 
method and apparatus for displaying images (e.g. 
motion pictures) corresponding to voice data or docu- 
ment data, and more particularly, to an image display 
method and apparatus capable of displaying sign lan- 
guage images corresponding to speech. 

The addition of voice data (such as a voice, music 
and the like) to video images has been long per- 
formed, and an apparatus which not only reproduces 
video images but generates sounds related to the im- 
ages is well known. 

However, there has been provided no apparatus 
which generates and displays images corresponding 
to input document data, voice signals and the like. For 
example, the contents of a television program are, as 
is well known, given to deaf persons by superimpos- 
ing a caption on the screen or displaying the contents 
of speech in sign language on a part of the screen. 
However, since such caption and sign language dis- 
plays incur much trouble and cost in editing, they are, 
as matters now stand, difficult to carry out. 

Furthermore, the provision of information in char- 
acters, such as a caption, requires the deaf persons 
to make efforts toward understanding the information 
and to concentrate their attention on reading the 
characters. 

It is an object of the present invention to provide 
an image display method and apparatus capable of 
displaying images corresponding to document data or 
voice data. 

In order to achieve the above object, according to 
one aspect of the present invention, there is provided 
an image display apparatus comprising image stor- 
age means for storing image data, storage means for 
storing document data, readout means for reading out 
document data stored in the storage means and read- 
ing out image data corresponding to a character 
string of the document data from the image storage 
means, and display means for displaying the image 
data read out by the readout means. 

The above structure makes it possible to display 
images corresponding to document data. 

Since document data is displayed in a character 
pattern together with image data, the character pat- 
tern and the image data can be ascertained in corre- 
spondence. 

Since image data to be displayed may be sign 
language image data corresponding to a word or a 
phrase, an easily recognizable image can be provid- 
ed. 

Since image data to be displayed may be a mo- 
tion picture, an easily recognizable image can be pro- 
vided. 

The addition of an expansion means for expand- 
ing image data allows display image data to be stored 
under compression, thereby taking less memory. 

According to another aspect of the present inven- 
tion, there is provided an image display apparatus 



comprising image storage means for storing image 
data, speech input means for inputting speech there- 
from, speech recognizing means for recognizing the 
speech inputted from the speech input means and 

5 generating a character code string corresponding to 
the recognized speech, readout means for reading 
out image data corresponding to the character code 
string from the image storage means, and display 
means for displaying the image data read out by the 

10 readout means. 

Therefore, it is possible to generate images cor- 
responding to speech. 

According to still another aspect of the present in- 
vention, there is provided an image display method 

15 comprising the steps of inputting speech and recog- 
nizing the speech, and displaying an image corre- 
sponding to the speech based on the result of the 
speech recognition. 

Therefore, it is possible to display images corre- 

20 sponding to speech. 

Steps may be added for inputting a video signal 
together with speech, and compositely displaying an 
image corresponding to the video signal and the im- 
age corresponding to the recognized speech. 

25 Other objects and features of the present inven- 

tion will become clear upon reading the following de- 
scription. 

BRIEF DESCRIPTION OF THE DRAWINGS 

30 

Fig. 1 is a block diagram showing the general 
structure of an image display apparatus accord- 
ing to a first embodiment of the present invention; 
Fig. 2 is a flow chart explaining processing of the 
35 image display apparatus according to the first 

embodiment; 

Fig. 3 is a block diagram showing the general 
structure of an image display apparatus accord- 
ing to a second embodiment of the preset inven- 
40 tion; 

Fig. 4 is a block diagram showing the general 
structure of an image display apparatus accord- 
ing to a third embodiment of the present inven- 
tion; and 

45 Fig. 5 is a display example of the image display 

apparatus according to the third embodiment of 
the present invention. 

DESCRIPTION OF THE PREFERRED 
50 EMBODIMENTS 

Preferred embodiments of the present invention 
will now be described with reference to the drawings. 

55 [Embodiment 1] 

Fig. 1 is a block diagram showing the general 
structure of an image display apparatus according to 
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a first embodiment of the present invention. 

Referring to Fig. 1, a CPU 1 controls the whole 
apparatus, and a ROM/RAM 2 comprises a ROM in 
which a control program of the CPU 1 and various 
kinds of data are stored, and a RAM which is used as 5 
a work area of the CPU 1 and temporarily stores va- 
rious kinds of data therein. A direct memory access 
(DMA) controller (DMAC) 3 is used to input and output 
data by DMA in the ROM/RAM 2, between the 
ROM/RAM 2 and an input-output 4, or among a char- 10 
acter information storage device 5, an image diction- 
ary storage device 6 and an image expansion device 
7. The input-output 4 controls the interface between 
a keyboard 1 0 (which is operated by an operator to in- 
put document data, commands and the like therein) 15 
and unillustrated peripheral devices. Text inputted 
through the input-output 4 is stored in the character 
information storage device 5 which stores text data in 
character codes and the like. The image dictionary 
storage device 6 stores compressed image informa- 20 
tion corresponding to character information (words, 
phrases and so on). The image information includes, 
for example, sign language images (motion pictures) 
corresponding to words, phrases and so on. The im- 
age expansion device 7 expands and outputs the im- 25 
age information compressed and stored in the image 
dictionary storage device 6. An image output signal 
processor 8 outputs the expanded image information 
to a display 9, such as a CRT display. Numeral 10 de- 
notes a CPU bus. 30 

In the above structure, character information 
(character codes) stored in the character information 
storage device 5 is read out under the control of the 
CPU 1 , and image data stored in the image dictionary 
storage device 6 is accessed corresponding to the 35 
character codes. In this case, the image dictionary 
storage device 6 stores sign language images corre- 
sponding to the character information; that is, words 
read out from the character information storage de- 
vice 5. The sign language images may be obtained, 40 
for example, by adding motion to images produced by 
computer graphics or actually photographing per- 
sons. 

The image data stored in the image dictionary 
storage device 6 is generally compressed because of 45 
its massive amount. In order to compress the image 
data, for example, the standard of Comite Consultatif 
Internationale Telegraphique et Telephonique H. 261 , 
Moving Picture Image Coding Experts Group and the 
like for compressing motion pictures may be used. so 
The image data corresponding to the character infor- 
mation is transmitted to the image expansion device 
7 by the DMAC 3. The compressed image data is ex- 
panded by the image expansion device 7, converted 
into video signals by the image output signal proces- 55 
sor 8 and displayed on the display 9. 

Af low chart of Fig. 2 explains the above process. 
A control program for carrying out the process is stor- 



ed in the ROM/RAM 2. 

Document data or various phrases are stored in 
the character information storage device 5, and the 
process is started, for example, in response to input 
of a command to read out the document data from the 
character information storage device 5 through the 
keyboard 10. First, the document data is read out 
from the character information storage device 5 in 
Step S1, image data corresponding to the read docu- 
ment data, such as character strings, words, phrases 
and the like, are read out from the image dictionary 
storage device 6 in Step S2, and the read image data 
(compressed data) is outputted to the image expan- 
sion device 7 in Step S3. The correspondence be- 
tween the document data and the image data is de- 
termined, for example, by reading out storage ad- 
dresses of the image data corresponding to the docu- 
ment data with reference to a table or the like stored 
in the ROM/RAM 2. In Step S4, the expanded image 
data is displayed on the display 9. 

The steps S1 to S4 are sequentially carried outfor 
character strings, words or phrases constituting the 
text stored in the character information storage de- 
vice 5, and obtained images (motion pictures) are se- 
quentially displayed on the display 9, thereby allowing 
understanding of the text in successive sign language 
images. 

As described above, according to the first em- 
bodiment, image data corresponding to document 
data (for example, sign language images correspond- 
ing to speech) can be outputted and displayed. 

Furthermore, if a character pattern generator for 
generating a character pattern corresponding to a 
character code is coupled to bus 10, it is possible to 
display in a part of the display 9 the text stored in the 
character information storage device 5 in character 
patterns while displaying sign language images cor- 
responding to the character patterns. This configura- 
tion makes it possible to practice sign language and 
ascertain which part of the text is being displayed in 
sign language. 

[Embodiment 2] 

Fig. 3 is a block diagram showing the general 
structure of an image display apparatus according to 
a second embodiment of the present invention. Com- 
ponents common to Fig. 1 are denoted by like numer- 
als, and the description thereof is omitted. 

Referring to Fig. 3, a voice signal inputted from a 
microphone input terminal 11 is amplified by an am- 
plifier 12, inputted to a speech recognizer 13 to be 
recognized, and converted into a character code with 
reference to a speech dictionary 14. A storage device 
1 5 stores the voice data recognized in the speech rec- 
ognizer 13. 

According to the above structure, the voice sig- 
nal inputted from the microphone input terminal 11 is 
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amplified by the amplifier 12, and recognized by the 
speech recognizer 13. The recognized voice data is 
converted into character data and stored in the stor- 
age device 15. The character data stored in the stor- 
age device 15 is read out by a CPU 1a, and image 
data stored in an image dictionary 6 is accessed in 
correspondence with the read-out character data. 
The image data related to the character data is trans- 
mitted to an image expansion device 7 by DMA under 
the control of a DM AC 3 in the same manner as de- 
scribed above. The compressed image data is ex- 
panded by the image expansion device 7, converted 
into video signals by an image output signal proces- 
sor 8, and displayed on a display 9. 

In this case, character data is sequentially read 
out from the storage device 15 and image data cor- 
responding to the read-out character data Is dis- 
played on the display 9 in the same manner as in the 
above flow chart of Fig. 2. 

As described above, according to the second em- 
bodiment, after an input voice signal is subjected to 
speech recognition, an image dictionary related to 
the recognized voice signal (for example, a sign lan- 
guage image) is read out and displayed on the dis- 
play, thereby making it possible to convert and display 
voice data into sign language images in real time. 

[Embodiment 3] 

Fig. 4 is a block diagram showing the general 
structure of an image display apparatus according to 
a third embodiment of the present invention. Compo- 
nents common to Figs. 1 and 3 are denoted by like nu- 
merals and the description thereof is omitted. Refer- 
ring to Fig. 4, an input terminal 21 for inputting a video 
signal, a video input signal processor 22, an image 
composer 23 and a video output signal processor 24 
are added to the structure shown in Fig. 3. 

As described in the above second embodiment, 
a voice signal inputted from a microphone input ter- 
minal 11 is recognized and stored in a storage device 
1 5. After that, a compressed image related to the rec- 
ognized voice signal is read out from an image dic- 
tionary storage device 6. The compressed image is 
further expanded by an image expansion device 7, 
and then transmitted to the image composer 23 
through the video output signal processor 24. 

On the other hand, separately from the image 
corresponding to the voice signal, a video signal in- 
putted from the video input terminal 21 is transmitted 
to the image composer 23 through the video signal in- 
put processor 22. The image composer 23 composes 
the inputted video signal and the image read out from 
the image dictionary storage device 6 (expanded), 
and outputs a composite on a display 9. 

If speech inputted from the microphone input ter- 
minal 11 and a video signal inputted from the video in- 
put terminal 21 are in synchronization with each other 



in the structure shown in Fig. 4 (for example, a tele- 
vision program with video), since an image outputted 
from the video output signal processor 24 may lag be- 
hind a video signal outputted from the video input sig- 

5 nal processor 22, a delay circuit (not shown) may be 
provided for delaying the video signal inputted from 
the video signal input terminal 21 . 

Fig. 5 illustrates an example of a composite im- 
age obtained in the third embodiment. Video signals 

10 inputted from the video input terminal 21 and sign lan- 
guage images read out from the image dictionary 
storage device 6 (expanded) are displayed as a com- 
posite picture on a main screen 501 and a subscreen 
502 of the display 9. 

15 As described above, according to the third em- 

bodiment of the present invention, a sign language 
image corresponding to speech inputted from, for ex- 
ample, a microphone and video signals inputted from 
the video input terminal 21 are displayed in composite 

20 form, and a television program or the like is displayed 
on the main screen 501 while displaying sign lan- 
guage images corresponding to the sound of the tel- 
evision program on the subscreen 502. Therefore, 
deaf persons can also enjoy the television program. 

25 In the above embodiments, a speech synthesis 

device may be added for synthesizing speech input- 
ted from the microphone input terminal 11 with infor- 
mation stored in the storage devices 5 and 15. In ad- 
dition, a speaker or the like may be added in order to 

30 output speech as well as images at the display 9. 

The present invention may be applied to a system 
constituted by a plurality of input and output devices, 
or one apparatus comprising one such device. Need- 
less to say, the present invention may be applied to 

35 a case in which the system or the apparatus is pro- 
vided with a program for carrying out the present in- 
vention. 

The individual components shown in outline or 
designated by blocks in the Drawings are all well- 

40 known in the image display arts, and their specific 
construction and operation are not critical to the op- 
eration or best mode for carrying out the invention. 

While the present invention has been described 
with respect to what is presently considered to be the 

45 preferred embodiments, it is to be understood that the 
disclosure is not limited to these particular embodi- 
ments. To the contrary, the present disclosure in- 
cludes various modifications and equivalent arrange- 
ments included within the spirit and scope of the ap- 

50 pended claims. The scope of the following claims is 
to be accorded the broadest interpretation so as to en- 
compass all such modifications and equivalent struc- 
tures and functions. Features of the different em- 
bodiments described above may also be combined as 

55 desired. 
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Claims 

1. An image display apparatus, comprising: 

image storage means for storing image 
data corresponding to sign language information; 

document storage means for storing docu- 
ment data; 

readout means for reading out document 
data from said document storage means, and for 
reading out from said image storage means im- 
age data corresponding to a character string of 
the document data read out from said document 
storage means; and 

display means for displaying the image 
data read out by said readout means. 

2. An image display apparatus according to claim 1 , 
further comprising: 

control means for causing said display 
means to display the document data read out by 
said readout means in a character pattern togeth- 
er with the image data read out by said readout 
means, 

wherein the image data comprises sign 
language image data corresponding to a word or 
a phrase. 

3. An image display apparatus according to claim 1 , 
wherein said display means displays image data 
comprising a motion picture. 

4. An image display apparatus according to claim 1 , 
wherein said display means includes expansion 
means for expanding the image data read out by 
said readout means. 

5. An image display apparatus, comprising: 

image storage means for storing image 

data; 

speech input means for inputting speech; 

speech recognizing means for recognizing 
speech inputted by said speech input means, and 
for generating a character code string corre- 
sponding to the recognized speech; 

readout means for reading out from said 
image storage means image data corresponding 
to the character code string generated by said 
speech recognizing means; and 

display means for displaying the image 
data read out by said readout means. 

6. An image display apparatus according to claim 5, 
wherein said display means displays image data 
comprising sign language image data corre- 
sponding to a word or a phrase. 

7. An image display apparatus according to claim 5, 
wherein said display means displays image data 



comprising a motion picture. 

8. An image display apparatus according to claim 5, 
wherein said display means includes expansion 

5 means for expanding the image data read out by 

said readout means. 

9. An image display method, comprising the steps 
of: 

10 receiving and recognizing speech; and 

displaying an image corresponding to the 
recognized speech, the displayed image corre- 
sponding to sign language information. 

15 10. An image display method according to claim 9, 
further comprising the steps of: 

receiving image information together with 
the received speech; and 

displaying the received image information 
20 in a character pattern with the sign language in- 

formation. 

11. An image display method according to claim 9, 
wherein the displayed image comprises sign Ian- 

25 guage picture information. 

12. An image display method according to claim 9, 
wherein the displayed image comprises sign lan- 
guage motion picture information. 

30 

13. An image display method according to claim 9, 
further comprising the steps of: 

storing image data corresponding to 
speech; 

35 reading out the stored image data in accor- 

dance with the recognized speech; and 

expanding the read-out image data prior to 
displaying the image. 

40 14. An image display method according to claim 13, 
further comprising the steps of: 

receiving a video signal together with the 
speech; and 

displaying together an image of the video 
45 signal and the sign language information image. 

15. An image display apparatus having the features 
of any combination of the preceding claims. 

50 1 6. An image signal conveying images generated au- 
tomatically from document or speech signals by 
an apparatus according to any preceding claim. 

1 7. A record carrier conveying images generated, au- 
55 tomatically in response to document and/or 

speech data, by an apparatus according to any of 
claims 1 to 15. 
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(54) Method and apparatus for displaying sign language images corresponding to text or 
speech 

(57) In order to provide an image display method and 
apparatus capable of displaying images corresponding 
to document data or speech data, image data (such as 
sign language images) are stored in an image dictionary 
in motion picture form. Document data is read out from 
a character information storage device (or speech data 
is received), and a sign language image corresponding 
to a character string of the document data (or the speech) 
is selected from the image dictionary and displayed on 
a display. 
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